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DETAILED ACTION 

Response to Amendment 

1 . The Examiner acl<nowledges tine amended claims filed on October 24, 201 0. 
Claims 1, 40 and 44 have been amended. Claims 8-10, 21-39, 41, 43, 45 and 48-59 

have been cancelled. 

Response to Arguments 

2. Applicant's arguments with respect to independent claims 1 , 40 and 44 have 
been considered but are moot in view of the new grounds of rejection. 

3. Applicant's arguments filed October 24, 2010 have been fully considered but they 
are not persuasive. The Applicant further argues the following: 

a. Applicant has reviewed Metcalfe, Takahashi, Mann, and Matsumoto, and 
cannot locate this added subject matter in any of these references. As a first 
example, Takahashi suggests that portions of the image signal having saliency 
signal values below a threshold are simply discarded (see, e.g., para. [0094]). As 
a second example, Metcalfe suggests that portions of the image signal having 
saliency signal values below a threshold are not output (see, e.g., bottom of p. 6, 
through top of p. 7). As a third example, Mann suggests that portions of the 
image signal having saliency signal values below a threshold are just not 
recorded in the first place (see, e.g., col. 12, 11. 40-64). As a fourth example, 
Matsumoto suggests that portions of the image signal having saliency signal 
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values below a threshold are recorded at a greater compression level (see, e.g., 
col. 3, 1. 15, through col. 4, 1. 19). 

Therefore, none of the prior art of record suggests that the portions of the 
image signal having saliency signal values below a threshold are played back in 
an accelerated manner, in contradistinction to the claimed invention, as 
amended. As such, no combination of the prior art of record rises to the level of 
suggesting the subject matter of the independent claims, as amended. Applicant 
thus respectfully submits that the independent claims, as amended, are 
patentable over any combination of the cited prior art of record. 

> The Examiner would like to point out that in page 5, lines 20-22, Metcalfe 
teaches the following: 

"For example, areas of the recorded tape flagged as being of a low 
interest are played back relatively quickly, whilst areas of relatively high interest 
are played back at normal speeds". 

This teaches replaying the pictures such that the pictures having saliency 
signal signals below a threshold are played back in an accelerated manner for 
accelerated viewing thereof as claimed. 
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Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the phor art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary sl^ill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 44, 46 and 47 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Takahashi, US 2002/0041757 in view of Metcalfe, AU 743216 B. 

6. Regarding claim 44, Takahashi discloses an imaging system comprising an 
electronic camera {See figs. 4(a), 4(b), 8, 9(a), 9(b), 10, and 11) to produce an image 
signal representative of a viewed scene, a physically or mechanically operable user 
control {Takahashi discloses the use of buttons 109 to set auxiliary information (which 
the Examiner is interpreting as the saliency information) related to the image data being 
captured; page 6, ^0106. Takahashi further discloses the use of a pressure sensor 
109a and sweat sensor 109b to determine the auxiliary information related to the image 
data (page 6, ^0112- page 7, U 0114) to receive an input from a user and generate a 
first saliency signal while the image signal is being produced {Takahashi further 
discloses that the auxiliary information includes infonmation related to the persons (i.e. 
son, daughter, friend, father, mother) (see figs. 23(a) and 23 (b); page 8,^0117- page 
9, H 0124)) to receive an input from a user and to generate, in response to the input 
from the user a saliency signal (auxiliary information having information such as 
persons information)), saliency circuitry to automatically generate an image related 
second saliency signal in response to the image signal {Takahashi further teaches that 
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the auxiliary information includes a degree of importance of said persons appearing in 
the image data, and that said degree of importance can be determined based on the 
time length of a scene where a particular person set by the user appear; see page 8, H 
01 19. Takahashi further teaches that the level of importance of a particular scene 
would have a value between 0(0 = not important) and 1 (1 = most important) with 
increments of 0. 1 that can be determined based on the excitement of the user by using 
a pressure sensor of a sweat sensor or by having a measurement of loudness in the 
scene being captured (See ^0119). Takahashi further discloses that the level of 
importance which can be set manually could also be set automatically by use of said 
pressure, sweat or loudness sensors (Also in H 0119)), and circuitry to combine said first 
and second saliency signals while the image signal is being produced to provide a 
composite saliency signal {Takahashi discloses that the auxiliary information has 
persons information and degree of importance of said persons and that the auxiliary 
information is stored with the image signal and further discloses that the camera would 
also display the auxiliary information on a display screen (As shown in figs. 23, the user 
would set the auxiliary infonmation and the set auxiliary information would be displayed 
in the display as shown in figs. 25 and 26 (see persons information and the degree of 
importance displayed on the display); page 8, ^0118- page 9, H 0124)) (This teaches 
the combination of the two saliency signals while the image signal is being produced 
(the persons infonmation and the degree of importance information stored/displayed 
together in association with the image while the image is being captured upon operation 
of the user) to create a composite saliency signal as claimed. Therefore, by teaching 
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that the person's information and the degree of importance information is 
stored/displayed together in association with the image, Takahashi discloses circuitry 
for combining said first and second saliency signals while the image signal is being 
produced to provide a composite saliency signal as claimed), wherein tine first saliency 
signal, the second saliency signal, and the composite saliency signal are to indicate an 
amount of user interest in the viewed scene (Considering that the user sets the first 
saliency signal and the second saliency signal {As discussed above, information related 
to the persons (i.e. son, daughter, friend, father, mother) and the level of importance of 
a particular scene would have a value between 0(0 = not important) and 1 (1 = most 
important) with increments of 0. 1 that can be determined based on the excitement of the 
user by using a pressure sensor of a sweat sensor or by having a measurement of 
loudness in the scene being captured (See H 0119) and as shown in figs. 25 and 26, the 
composite saliency signal that is displayed on the screen having the first and second 
saliency signals combined to illustrate the level of importance of the scene being 
produced (page 8, ^0118- page 9, H 0124)), and the composite saliency signal is to be 
used to control operation of at least a part of the imaging system {Takahashi discloses 
that the auxiliary information has persons information and degree of importance of said 
persons and that the auxiliary information is stored with the image signal and further 
discloses that the camera would also display the auxiliary information on a display 
screen (As shown in figs. 23, the user would set the auxiliary information and the set 
auxiliary information would be displayed in the display as shown in figs. 25 and 26 (see 
persons information and the degree of importance displayed on the display); page 8, H 
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0118- page 9, H 0124)) being arranged to be controlled in response to the composite 
saliency signal (based on the auxiliary infonmation the display would display the 
auxiliary information as shown in figs. 25 and 26). 

Takahashi does not explicitly disclose a replay circuitry to replay the image signal 
such that portions of the image signal having saliency signal values below a threshold 
are played back in an accelerated manner for accelerated viewing thereof. 

However, Metcalfe discloses a camera apparatus {See fig. 1) comprising; 

an electronic camera {See fig. 1) to produce an image signal; 

a first user operable control {111 as shown in fig. 1) to selectively activate the 
electronic camera to take pictures; 

a second user operable control {112 as shown in fig. 1, note that the limitations 
"an additional physically or mechanically operable user control" are written as optional 
elements by using the word "or') to receive an input from a user and to generate, in 
response to the input from the user a saliency signal to indicate user interest in a picture 
{Metcalfe discloses generating a level of interest (LOI) set by the user when using 
button 112), the saliency signal to have at least one of (a) a value selected from at least 
three different discrete values {Metcalfe discloses assigning a level of interest to the 
image data being recorded, wherein the user can variably assign a plurality of level of 
interest through the capture of the video sequence; see Fig. 3. This teaches generating 
a saliency signal that is selected from at least three different discrete values while the 
image signal is being produced), or (b) a value selected fro a continuous range of 
values {Metcalfe discloses assigning a level of interest to the image data being 
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recorded, wherein the user can variably assign a plurality of level of interest through the 
capture of the a video sequence; see Fig. 3. This teaches generating a saliency signal 
that has a value selected fro a continuous range of values); 

a circuit to record the value of the saliency signal based on the input received via 
the second control contemporaneously with activation of the first control {Metcalfe 
discloses that the LOI is stored as the frames are captured by the video camera 101 
(Page 5, lines 8-35) and also that after operation of the record button 111, the LOI 
signals can be input by the user and recorded by the video camera (Page 4, lines 5-32). 
This inherently teaches a circuitry to record the value of the saliency signal based on 
the input received via the second control contemporaneously with activation of the first 
control as claimed since a circuitry is necessary to perform the recording operation of 
the LOI to the tape and also considering that the operation to record LOI signals can be 
performed right after the record button to capture image frames is operated as taught in 
Metcalfe. It is also noted that the word "contemporaneous" is defined as "Originating, 
existing, or happening during the same period of time" (Definition from "The American 
Heritage® Dictionary of the English Language", Fourth Edition, 2000)); 

replay circuitry to replay the image signal such that portions of the image signal 
having saliency signal values below a threshold are played back in an accelerated 
manner for accelerated viewing thereof {In page 5, lines 20-22, Metcalfe teaches "For 
example, areas of the recorded tape flagged as being of a low interest are olaved back 
relatively quickly, whilst areas of relatively high interest are played back at normal 
speeds ". This teaches the use of a replay circuit to replay the pictures such that the 
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pictures having saliency signal signals below a threshold are played back in an 
accelerated manner for accelerated viewing thereof as claimed), 

a memory to store the image signal and the saliency signal {video cassette120 
as shown in fig. 1; page 4, lines 5-13; page 5, lines 4-35), wherein at least one 
operation of the apparatus is controlled based on the saliency signal {Metcalfe discloses 
setting the LOI while the camera is activated to take pictures, and storing said LOI 
associated with the images so that said LOI can be used to control the reproduction of 
the images (i.e. printing, creating thumbnail files for photo albums, etc.)) being arranged 
to be controlled in response to the saliency signal (as discussed in page 6, line 31 - 
page 7, line 8 , Metcalfe discloses that the saliency signal (LOI) can be used to control 
the reproduction of the images (i.e. printing, creating thumbnail files for photo albums, 
etc.) (See page 4, line 5 - page 5, line 23; page 6, line 23 - page 7, line 8). 

Therefore, taking the combined teaching of Takahashi in view of Metcalfe as a 
whole, after considering the concept of replaying portions of a recorded video quicker 
when an associated level of interest is low and replaying portions of the video with high 
level of interest at normal speed as taught in Metcalfe, it would have been obvious to 
one of an ordinary skill in the art at the time the invention was made to modify the 
teaching of Takahashi to include replay circuitry to replay the image signal such that 
portions of the image signal having saliency signal values below a threshold are played 
back in an accelerated manner for accelerated viewing thereof. The motivation to do so 
would have been to allow the user to go through the most interesting parts of videos 
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that would help the user to edit the video as suggested by Metcalfe {Page 1, line 24 - 
page 2, line 6). 

7. Regarding claim 46, Takahashi discloses a user operable picture taking control 
to enable the electronic camera to take pictures {shooting button 104 as sliown in fig. 4 
(b)). 

8. Regarding claim 47, Takahashi discloses that the first saliency signal is to 

include more than two values {As shown in figs. 23 (a) and 23 (b), Takahashi discloses 
that more than one person can be selected (i.e. the son and the daughter); page 8, H 
0719). 

9. Claims 1, 3-5 and 11-20 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Metcalfe, AU 743216 B in view of Takahashi, US 2002/0041757 
A1. 

10. Regarding claim 1, Metcalfe discloses a camera apparatus {See fig. 1) 
comprising; 

an electronic camera {See fig. 1) to produce an image signal; 

a first user operable control {111 as shown in fig. 1) to selectively activate the 
electronic camera to take pictures; 

a second user operable control {112 as shown in fig. 1, note that the limitations 
"an additional physically or mechanically operable user control" are written as optional 
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elements by using the word "or') to receive an input from a user and to generate, in 
response to the input from the user a saliency signal to indicate user interest in a picture 
(Metcalfe discloses generating a level of interest (LOI) set by the user when using 
button 112), the saliency signal to have at least one of (a) a value selected from at least 
three different discrete values {Metcalfe discloses assigning a level of interest to the 
image data being recorded, wherein the user can variably assign a plurality of level of 
interest through the capture of the video sequence; see Fig. 3. This teaches generating 
a saliency signal that is selected from at least three different discrete values while the 
image signal is being produced), or (b) a value selected fro a continuous range of 
values {Metcalfe discloses assigning a level of interest to the image data being 
recorded, wherein the user can variably assign a plurality of level of interest through the 
capture of the a video sequence; see Fig. 3. This teaches generating a saliency signal 
that has a value selected fro a continuous range of values); 

a circuit to record the value of the saliency signal based on the input received via 
the second control contemporaneously with activation of the first control {Metcalfe 
discloses that the LOI is stored as the frames are captured by the video camera 101 
(Page 5, lines 8-35) and also that after operation of the record button 111, the LOI 
signals can be input by the user and recorded by the video camera (Page 4, lines 5-32). 
This inherently teaches a circuitry to record the value of the saliency signal based on 
the input received via the second control contemporaneously with activation of the first 
control as claimed since a circuitry is necessary to perform the recording operation of 
the LOI to the tape and also considering that the operation to record LOI signals can be 
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performed right after the record button to capture image frames is operated as taught in 
Metcalfe. It is also noted that the word "contemporaneous" is defined as "Originating, 
existing, or happening during the same period of time" (Definition from "The American 
Heritage® Dictionary of the English Language", Fourth Edition, 2000)); 

replay circuitry to replay the Image signal such that portions of the image signal 
having saliency signal values below a threshold are played back in an accelerated 
manner for accelerated viewing thereof {In page 5, lines 20-22, Metcalfe teaches "For 
example, areas of the recorded tape flagged as being of a low interest are played back 
relatively quickly, whilst areas of relatively high interest are played bacl< at normal 
speeds ". This teaches the use of a replay circuitry to replay the image signal such that 
portions of the image signal having saliency signal values below a threshold are played 
back in an accelerated manner for accelerated viewing thereof as claimed), 

a memory to store the image signal and the saliency signal {video cassette120 
as shown in fig. 1; page 4, lines 5-13; page 5, lines 4-35), wherein at least one 
operation of the apparatus is controlled based on the saliency signal {Metcalfe discloses 
setting the LOI while the camera is activated to take pictures, and storing said LOI 
associated with the images so that said LOI can be used to control the reproduction of 
the images (i.e. printing, creating thumbnail files for photo albums, etc.)) being arranged 
to be controlled in response to the saliency signal (as discussed in page 6, line 31 - 
page 7, line 8 , Metcalfe discloses that the saliency signal (LOI) can be used to control 
the reproduction of the images (i.e. printing, creating thumbnail files for photo albums, 
etc.) (See page 4, line 5 - page 5, line 23; page 6, line 23 - page 7, line 8). 
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Metcalfe does not explicitly disclose that the at least one operation is different 
from recording the value of the saliency signal in the memory. 

However, Takahashi discloses an electronic camera {See figs. 4(a), 4(b), 8, 9(a), 
9(b), 10, and 11) producing image signal, comprising a first user operable control 
{shooting button 104 as sliown in fig. 4 (b)) to selectively activate the electronic camera 
to take pictures, and a second user operable control {Takatiaslii discloses thie use of 
buttons 109 to set auxiliary infonvation (which the Examiner is interpreting as the 
saliency information) related to the image data being captured; page 6, H 0106. 
Takahashi further discloses the use of a pressure sensor 109a and sweat sensor 109b 
to determine the auxiliary information related to the image data (page 6, ^0112- page 
7, H 0114). Also Takahashi discloses that the auxiliary information includes information 
related to the persons (i.e. son, daughter, friend, father, mother) and a degree of 
importance of said persons appearing in the image data (see figs. 23(a) and 23 (b); 
page 8, ^0117 - page 9, H 0124)) to receive an input from a user and to generate, in 
response to the input from the user a saliency signal to indicate user interest in a picture 
{auxiliary information having information such as persons information and degree of 
importance of the persons in the image), the saliency signal to have at least one of (a) a 
value selected from at least three different discrete values (as shown in figs. 23(a) and 
23(b), the persons information and degree of importance of the persons in the image 
can change in value between a plurality of values (i.e. three or more values as 
claimed)), or (b) a value selected from a continuous range of values {the auxiliary 
infomnation can be changed during the capture of the image signal to a plurality of 



Application/Control Number: 10/696,567 Page 14 

Art Unit: 2622 

continuous values as shown in figs. 23(a) and 23(b)), and a memory {Fig. 1: 13) 
arranged to store the image signal and the saliency signal (Takahashi further teaches 
storing the auxiliary information in the header of the scene; page 8,^0123- page 9, H 
0126), wherein at least one operation of the apparatus is controlled based on the 
saliency signal {Takahashi discloses recording the auxiliary information and further 
discloses that the camera would also display the auxiliary information on a display 
screen (As shown in figs. 23, the user would set the auxiliary information and the set 
auxiliary infomiation would be displayed in the display as shown in figs. 25 and 26; page 
8, ^0118- page 9, ^ 0124). It is noted that by teaching that based on the auxiliary 
information the display would display the auxiliary information Takahashi discloses that 
at least one operation of the apparatus is controlled based on the saliency signa\), the at 
least one operation being different from to recording the saliency signal in the memory 
{The Examiner is interpreting displaying the auxiliary information using the display of the 
camera as the operation that is different from recording the saliency signal in the 
memory as claimed). Displaying the auxiliary information while capturing the image 
data is advantageous because it would provide the user with a user friendly interface 
that would allow changing the degree of importance of the images being captured and 
would also allow the user to be aware of the information being added to the image data. 

Therefore, taking the combined teaching of Metcalfe in view of Takahashi as a 
whole, it would have been obvious to one of an ordinary skill in the art at the time the 
invention was made to apply the concept of having the camera further displaying the 
salient signal being assign to the image data as taught in Takahashi to modify the 
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teaching of Metcalfe to perform an operation in response to tlie saliency signal, the 
operation being different from recording the saliency signal in the memory. The 
motivation to do so would have been to provide the user with a user friendly interface 
that would allow changing the degree of importance of the images being captured and 
would also allow the user to be aware of the information being added to the image data. 

1 1 . Regarding claim 3, the combined teaching of Metcalfe in view of Takahashi as 
discussed and analyzed in claim 1 further teaches a buffer to receive the image signal, 
the buffer having a capacity controlled by the value of the saliency signal {In a further 
embodiment, Takahashi discloses the concept of determining the amount of data to be 
transmitted based on the cost of transmission service; wherein the allowed length of 
data (L) is determined and compared to the amount of data to be transmitted, if the 
amount of data to be transmitted the apparatus would select video section with a priority 
higher than a threshold value. If after selecting those video with the priority higher than 
a threshold, the length of data still more than the allowed length, the apparatus would 
increase the priority threshold, however, if the length is less that the allowed length, the 
video signals are transmitted (Page 12, H 0151-0154). By teaching adjusting the 
amount of data to be transmitted based on the length allowed for transmission and also 
based on the priority of the video signals to be transmitted, Takahashi inherently 
discloses "a buffer to receive the image signal, the buffer having a capacity controlled 
by the value of the saliency signal" as claimed since the amount being adjusted in the 
apparatus needs to be in a particular buffer or memory prior to transmission in order to 
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properly select the video signals with higher priority). One of an ordinary skill in the art 
would have found obvious to apply the concepts of adjusting the amount of data to be 
transmitted based on the priority set to the video signals to further modify the camera 
apparatus to have a buffer for receiving said image signal, the buffer having a capacity 
arranged to be controlled by the value of the saliency signal during operation of the 
camera apparatus with the motivation of select as many as possible the most important 
video signals from the video data for transmission as suggested by Takahashi {Page 
12, H 0154). 

1 2. Regarding claim 4, the combined teaching of Metcalfe in view of Takahashi as 
discussed and analyzed in claim 1 further teaches image selection circuitry to receive 
the saliency and image signals and to selectively pass the image signal based on the 

saliency signal {As taught in Metcalfe, the camera receives the saliency signal (LOI) and 
based on said saliency signal, when reproducing, the camera would select particular 
images based on the degree of importance as set by the user when recording the 
images. By teaching that the images with a high LOI would be selected for creating an 
album or to be printed with high quality further teaches that images that do not have a 
high LOI would be passed or skipped since only the images with a high LOI would be 
displayed in the virtual album. Metcalfe as applied reads on ". . .image selection circuitry 
to receive the saliency and image signals and to selectively pass the image signal 
based on the saliency signal" as claimed since the display or skip of images is 
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determined based on the LOI set. (See pages 5-7, specifically page 6, line 31 - page 7, 
line 8)). 

1 3. Regarding claim 5, the combined teaching of Metcalfe in view of Takahashi as 

discussed and analyzed in claim 1 further teaches management circuitry to selectively 
retain in the memory {memory 13 in Takatiashi, fig. 1) images associated with higher 
saliency levels in preference to images with lower saliency levels {As discussed witli 
respect to claim 1, Takahashi further discloses that the images are recorded or 
transmitted based on the importance level of the image, wherein only images with high 
importance level can be recorded in order to reduce the amount of use of the recording 
medium; page 5, H 0094). It would have been obvious to one of an ordinary skill in the 
art at the time the invention was made to modify the teaching of Metcalfe with the 
concepts and teaching of Takahashi to further include management circuitry to 
selectively retain in the memory, images associated with higher saliency levels in 
preference to images with lower saliency levels. The motivation to do so would have 
been to reduce the amount of use of the recording medium as suggested by Takahashi 
{Page 5, H 0094). 

14. Regarding claim 1 1 , Metcalfe discloses that the first user control includes a 
normal picture taking control on the electronic camera {111 as shown in fig. 1). 
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15. Regarding claim 12, the combined teaching of Metcalfe in view of Tal<ahashi as 
discussed and analyzed in claim 1 further teaches that the saliency signal is a first 
saliency signal and further comprising a third user operable control to generate a 
second saliency signal {As shown in Takahashi, figs. 4(b): 109; fig. 9(a): 109, 10: 109a, 
and 11: 109b, Takahashi discloses the use of a plurality of buttons to select from 
different auxiliary information to be assigned to the video signal (note that the buttons 
109 are physically located on the camera). Furthermore, as shown in figs 23(a) and 
23(b), Takahashi further discloses that the auxiliary infonmation can be selected using a 
monitor, wherein the user can select the person information and the degree of 
importance of the persons using buttons 103m, 103n, 101m and 101 n; page 8, H 0119 
(although these buttons are displayed on the touch screen display, the Examiner 
understands that the buttons are considered different user controls since they have 
different physical location on said display)). Grounds for rejecting claim 1 apply here. 

16. Regarding claim 13, the combined teaching of Metcalfe in view of Takahashi as 
discussed and analyzed in claims 1 and 12 further teaches saliency circuitry to combine 

the first and second signals to form a complex saliency signal {the Examiner is reading 
the complex saliency signal as the combined information having the persons information 
and the degree of importance information as shown in Takahashi), the complex saliency 
signal being the saliency signal to control at least one operation {The Examiner is 
reading the operation as the display operation in the camera in Takahashi as discussed 
in claim 1) and the saliency signal stored by the memory {As discussed in claim 1, 
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Takahashi discloses that the auxiliary information has persons information and degree 
of importance of said persons and that the auxiliary infonmation is stored with the image 
signal and further discloses that the camera would also display the auxiliary information 
on a display screen (As shown in figs. 23, the user would set the auxiliary infonmation 
and the set auxiliary information would be displayed in the display as shown in figs. 25 
and 26 (see persons information and the degree of importance displayed on the 
display); page 8, ^0118- page 9, H 0124)) being arranged to be controlled in response 
to the saliency signal (based on the auxiliary information the display would display the 
auxiliary information as shown in figs. 25 and 26. Furthermore, the plurality of saliency 
signals are both stored in the memory and the display operation of the camera is 
performed based on the first and second saliency signals as they are input and stored in 
memory). Grounds for rejecting claim 1 and 12 apply here. 

1 7. Regarding claim 14, the combined teaching of Metcalfe in view of Tal<ahashi as 
discussed and analyzed in claim 1 further teaches that the saliency signal is a first 
saliency signal {As discussed in claim 1, the saliency signal in Metcalfe for controlling 
the level of interest of a captured scene is considered a first saliency signal as claimed. 
Furthermore, Tal<ahashi further teaches the use of a plurality of saliency signals to be 
used to control the operation of the camera as discussed in claim 1) and further 
comprising a saliency circuitry to generate an image related saliency signal in response 
to said image signal {Metcalfe discloses controlling the reproduction operation of the 
camera based on the associated saliency signal to the image signal so that when 
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reproducing the image signal witli liiglier importance would be displayed differently from 
the other image signal. See page 4, line 5 - page 5, line 23; page 6, line 23 - page 7, 
line 8. Takahashi further discloses reproducing the video based on the importance level 
of the image, wherein only images with high importance level can be reproduced so that 
the user can enjoy the recorded work without feeling tired, and the power consumption 
is reduced to secure more driving time; page 10, H 0131). 

18. Regarding claim 15, limitations have been discussed and analyzed in claim 13. 

19. Regarding claim 16, Metcalfe discloses circuitry to incorporate the saliency 
signal in the image signal {Metcalfe discloses incorporating said saliency signal (LOI) to 
each of the frames in the image signal. See page 4, line 5 - page 5, line 23; page 6, 
line 23 - page 7, line 8). 

20. Regarding claim 17, the combined teaching of Metcalfe in view of Takahashi as 
discussed and analyzed in claims 1 teaches that the second user control is part of the 

camera or is physically attached to the camera body {See Metcalfe, controls in Fig. 1; 
see also Takahashi buttons 109 as shown in figs. 4(b), 9(b), 10 and 11, and buttons 
103m, 103n, 101m, 101 n as shown in figs. 23(a) and 23(b)). 

21 . Regarding claim 18, the combined teaching of Metcalfe in view of Takahashi 
fails to teach that the second user control is a remote control for communication with the 
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camera. However, the Examiner takes Official Notice that the concept of controlling a 
camera with a remote control is well known in the art at the time the invention was made 
and that it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have made the user control of Metcalfe and Takahashi a remote 
control as opposed to a camera-body Integrated control. One would have been 
motivated to do so because it is well known in the art that by using a remote control to 
control some elements of a camera, the user does not have to be near the camera to 
send and receive desired signals from the camera. This is particularly advantageous in 
cases where plural cameras are used or cameras are placed out of the reach of the 
user (e.g. surveillance cameras), where the remote control would allow the user to send 
signals to the camera(s) from a separate location, thereby simplifying camera control for 
the user. It is noted that in the previous Office Action, claim 18 was rejected taking 
Official Notice to the recited limitations. Because the Applicant failed to traverse the 
Examiner's assertion of Official Notice, the well known in the art statement is taken to be 
admitted prior art. See MPEP § 2144.03 [R-1] (C). 

22. Regarding claim 19, l\1etcalfe discloses that the second user control comprises 
a physically movable control member {pressure button that assign a level of interest of 
an image signal based on tine pressure applied to said button) and a sensor arranged to 
be responsive to movement of the control member (See page 4, line 27 - page 5, line 
4). By teaching a pressure button that assigns a LOI of an image signal based on the 
pressure applied to the button, Metcalfe discloses a physically movable control member 
(in this case the pressure button as discussed in Metcalfe) and a sensor arranged to be 



Application/Control Number: 10/696,567 Page 22 

Art Unit: 2622 

responsive to movement of the control member since by teacliing that the LOI is 
assigned based on the pressure applied to the button, the use of a sensor to determine 
the applied pressure in order to assigned the LOI of an image is inherent and 
necessitated in Metcalfe). 

23. Regarding claim 20, the Examiner notes tliat the limitations "the second user 
control comprises a pressure or force sensing transducer for deriving the saliency 
signal that can have values that are continuously variable", the elements "pressure" or 
"force sensing transducer" are written as optional elements by using the word "or". 

Metcalfe discloses that the second user control comprises a pressure or force 
sensing transducer for deriving the saliency signal that can have values that are 
continuously variable {Metcalfe as applied to claim 19, teaches the use of pressure 
(pressure button that assign a level of interest of an image signal based on the pressure 
applied to said button; page 4, line 27 - page 5, line 4) to determine the value of the 
saliency signal. By teaching that Metcalfe discloses the use of pressure applied to the 
button, Metcalfe discloses the use of pressure to derive the value of the saliency signal 
as claimed. 
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24. Claims 2 and 7 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Metcalfe, AU 743216 B in view of Takahashi, US 2002/0041757 A1 and further 
in view of Matsumoto et al., US Patent 6,795,642 B2. 

25. Regarding claim 2, although Takahashi teaches a compression circuitry {See 
Takahashi, fig. 1: 15; page 5, H 0097; page 6, H 0105) to compress the image signals 
and that although the invention is described on the premise that a shot picture is 
recorded, a shot picture is not necessarily recorded, and it can be used also when 
compressed video and audio data are transmitted as they are to be used on a network 
or the like {page 10, H 0130), the combined teaching of Metcalfe in view of Takahashi 
fails to teach compressing the image signals to an extent determined by the saliency 
signal. 

However, Matsumoto et al. teaches the concept of having a video recording 

apparatus {Fig. 2) of a surveillance system, recording video data captured by an 
electronic camera {See fig. 1), wherein when an alarm is activated, the importance of 
the video is determined to be high as compared to when the alarm is not activated (Co/. 
3, lines 46-67). Matsumoto et al. further discloses that based on the degree of 
importance given to the video signal, the data compression is also adjusted {i.e. if the 
importance degree of the video is low, it would be compressed at high level and if the 
importance degree of the video is high, said video would be compressed at low level) 
{Col. 3, line 15- col. 4, lines 19). Matsumoto also discloses that the importance level 
can also be adjusted by the user operating the surveillance system {Col. 7, lines 22-34). 
Matsumoto et al. further discloses that by adjusting the compression of the video being 



Application/Control Number: 10/696,567 Page 24 

Art Unit: 2622 

captured, it is possible to record the monitoring image data having a high degree of 
importance as much as possible (Co/. 7, lines 35-43). 

Therefore, taking the combined teaching of Metcalfe in view of Takahashi and 
further in view of Matsumoto et al. as a whole, it would have been obvious to one of 
ordinary skill in the art a the time the invention was made to apply the concept of 
adjusting the compression of a video captured by an electronic camera based on a 
degree of importance assign to the video as discussed in Matsumoto et al. to modify the 
teaching of Metcalfe and Takahashi by compressing the image signals to an extent 
determined by the saliency signal. The motivation to do so would have been to record 
the monitoring image data having a high degree of importance as much as possible as 
suggested in Matsumoto et al. (Co/. 7, lines 35-43). 

26. Regarding claim 7, the combined teaching of Metcalfe in view of Takahashi and 
further in view of Matsumoto et al. further teaches a management circuitry to selectively 
retaining in the memory images associated with higher saliency levels in preference to 
images with lower saliency levels (Takahashi further discloses that the images are 

recorded or transmitted based on the importance level of the image, wherein only 
images with high importance level can be recorded in order to reduce the amount of use 
of the recording medium; page 5, H 0094). 
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27. Claims 40 and 42 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Metcalfe, AU 743216 B in view of Takahashi, US 2002/0041757 A1 and further 
in view of Matsumoto et al., US Patent 6,795,642 B2. 

28. Regarding claim 40, Metcalfe discloses an imaging system {See fig. 1) 
comprising: 

an electronic camera (See fig. 1) to produce an image signal; 

physically or mechanically operable user controls {See user controls 106, 104, 
111, 112, 110 and 108 as shown in fig. 1), the user controls being arranged to receive 
an input from a user and to generate first saliency signal {Metcalfe discloses the use of 
button 112 to generate a plurality of saliency signals (Level of interest signals "LOI") to 
be associated to the image signal being recorded with the camera) while the image 
signal is being produced, and 

saliency circuitry {the camera in Metcalfe inherently has a saliency signal circuitry 
to generate the saliency signal upon operation of the camera button 1 12) for storing 
said first saliency signal {Metcalfe discloses recording the plurality of saliency signals in 
a memory (tape 120 in fig. 1) in association with the image data; see page 4, line 5 - 
page 5, line 23; page 6, line 23 - page 7, line 8), the saliency signal having at least one 
of (a) a value selected from at least three different discrete values {Metcalfe discloses 
assigning a level of interest to the image data being recorded, wherein the user can 
variably assign a plurality of level of interest through the capture of the a video 
sequence; see Fig. 3. This teaches generating a saliency signal having a value from at 
least three different discrete values), or (b) a value selected from a continuous range of 
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values {Metcalfe discloses assigning a level of interest to the image data being 
recorded, wherein the user can variably assign a plurality of level of interest through the 
capture of the a video sequence; see Fig. 3. This teaches generating a saliency signal 
having a value selected from a continuous range of values), 

a memory {120 as shown in fig. 1) to store the image signal and the saliency 
signal in response to the saliency signal {page 4, lines 5-13; page 5, lines 4-35), and 
operation of at least part of the electronic camera being arranged to be controlled in 
response to the saliency signal (as discussed in page 6, line 31 - page 7, line 8, 
Metcalfe discloses that the saliency signal (LOI signal stored in memory 120) can be 
used to control the reproduction of the images (i.e. printing, creating thumbnail files for 
photo albums, etc.); and 

replay circuitry to replay the image signal such that portions of the image signal 
having saliency signal values below a threshold are played back in an accelerated 
manner for accelerated viewing thereof {In page 5, lines 20-22, Metcalfe teaches Tor 
example, areas of the recorded tape flaaaed as being of a low interest are played back 
relatively quickly, whilst areas of relatively high interest are played back at normal 
speeds ". This teaches a replay circuitry to replay the image signal such that portions of 
the image signal having saliency signal values below a threshold are played back in an 
accelerated manner for accelerated viewing thereof as claimed) (See page 4, line 5 - 
page 5, line 23; page 6, line 23 - page 7, line 8). 

Metcalfe does not explicitly disclose at least two physically or mechanically 
operable user controls, each of the user controls for generating first and second 
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saliency signals; that said saliency circuitry combines said first and second saliency 
signals to form a complex saliency signal; and that said memory stores the image signal 
in place of a stored image when the value of the saliency signal is greater than a value 
of a second saliency signal associated with the stored image and the memory is full. 

However, Takahashi teaches an imaging system comprising an electronic 
camera {See figs. 4(a), 4(b), 8, 9(a), 9(b), 10, and 11) to produce an image signal 
representative of a viewed scene, a physically or mechanically operable user control 
(Takahashi discloses the use of buttons 109 to set auxiliary information (which the 
Examiner is interpreting as the saliency information) related to the image data being 
captured; page 6, ^0106. Takahashi further discloses the use of a pressure sensor 
109a and sweat sensor 109b to determine the auxiliary information related to the image 
data (page 6, ^0112- page 7, H 0114) to receive an input from a user and generate a 
first saliency signal while the image signal is being produced {Takahashi further 
discloses that the auxiliary information includes information related to the persons (i.e. 
son, daughter, friend, father, mother) (see figs. 23(a) and 23 (b); page 8,^0117 - page 
9, H 0124)) to receive an input from a user and to generate, in response to the input 
from the user a saliency signal (auxiliary information having information such as 
persons information)), saliency circuitry to automatically generate an image related 
second saliency signal in response to the image signal {Takahashi further teaches that 
the auxiliary information includes a degree of importance of said persons appearing in 
the image data, and that said degree of importance can be determined based on the 
time length of a scene where a particular person set by the user appear; see page 8, H 
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01 19. Takahashi further teaches that the level of importance of a particular scene 
would have a value between 0(0 = not important) and 1 (1 = most important) with 
increments of 0. 1 that can be determined based on the excitement of the user by using 
a pressure sensor of a sweat sensor or by having a measurement of loudness in the 
scene being captured (See H 0119). Takahashi further discloses that the level of 
importance which can be set manually could also be set automatically by use of said 
pressure, sweat or loudness sensors (Also in H 0119)), and circuitry to combine said first 
and second saliency signals while the image signal is being produced to provide a 
composite saliency signal {Takahashi discloses that the auxiliary information has 
persons information and degree of importance of said persons and that the auxiliary 
information is stored with the image signal and further discloses that the camera would 
also display the auxiliary information on a display screen (As shown in figs. 23, the user 
would set the auxiliary information and the set auxiliary information would be displayed 
in the display as shown in figs. 25 and 26 (see persons information and the degree of 
importance displayed on the display); page 8, ^0118- page 9, H 0124)) (This teaches 
the combination of the two saliency signals while the image signal is being produced 
(the persons information and the degree of importance information stored/displayed 
together in association with the image while the image is being captured upon operation 
of the user) to create a composite saliency signal as claimed. Therefore, by teaching 
that the person's information and the degree of importance information is 
stored/displayed together in association with the image, Takahashi discloses circuitry 
for combining said first and second saliency signals while the image signal is being 
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produced to provide a composite saliency signal as claimed), wherein the first saliency 
signal, the second saliency signal, and the composite saliency signal are to indicate an 
amount of user interest in the viewed scene (Considering that the user sets the first 
saliency signal and the second saliency signal {As discussed above, information related 
to the persons (i.e. son, daughter, friend, father, mother) and the level of importance of 
a particular scene would have a value between 0(0 = not important) and 1 (1 = most 
important) with increments of 0. 1 that can be determined based on the excitement of the 
user by using a pressure sensor of a sweat sensor or by having a measurement of 
loudness in the scene being captured (See ^ 0119) and as shown in figs. 25 and 26, the 
composite saliency signal that is displayed on the screen having the first and second 
saliency signals combined to illustrate the level of importance of the scene being 
produced (page 8, ^0118- page 9, H 0124)), and the composite saliency signal is to be 
used to control operation of at least a part of the imaging system {Tal<ahashi discloses 
that the auxiliary information has persons information and degree of importance of said 
persons and that the auxiliary information is stored with the image signal and further 
discloses that the camera would also display the auxiliary infonmation on a display 
screen (As shown in figs. 23, the user would set the auxiliary information and the set 
auxiliary information would be displayed in the display as shown in figs. 25 and 26 (see 
persons information and the degree of importance displayed on the display); page 8, H 
0118- page 9, H 0124)) being arranged to be controlled in response to the composite 
saliency signal (based on the auxiliary information the display would display the 
auxiliary infomnation as shown in figs. 25 and 26). 
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Therefore, taking the combined teaching of Metcalfe in view of Takahashi as a 
whole, it would have been obvious to one of an ordinary skill in the art at the time the 
invention was made to apply the concept of having the camera with a plurality of 
operation controls to generate different saliency signals to be combined to control the 
operation of the display to further displaying the combined saliency signal being assign 
to the image data as taught in Takahashi to modify the teaching of Metcalfe to have at 
least two physically or mechanically operable user controls, each of the user controls for 
generating first and second saliency signals; that said saliency circuitry combines said 
first and second saliency signals to form a composite saliency signal; that the operation 
of at least part of the electronic camera being arranged to be controlled in response to 
the complex saliency signal. The motivation to do so would have been to provide the 
user with a user friendly interface that would allow changing the degree of importance of 
the images being captured and would also allow the user to be aware of the information 
being added to the image data. 

The combined teaching of Metcalfe in view of Takahashi fails to teach that the 
memory stores the image signal in place of a stored image when the value of the 
saliency signal is greater than a value of a second saliency signal associated with the 
stored image and the memory is full. 

However, Matsumoto et al. teaches the concept of having a video recording 
apparatus {Fig. 2) of a surveillance system, recording video data captured by an 
electronic camera (See fig. 1), wherein when an alarm is activated, the importance of 
the video is determined to be high as compared to when the alarm is not activated (Co/. 
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3, lines 46-67). Matsumoto et al. further discloses tliat based on the degree of 
importance given to the video signal, the data compression is also adjusted {i.e. iftiie 
importance degree of the video is low, it would be compressed at high level and if the 
importance degree of the video is high, said video would be compressed at low level) 
{Col. 3, line 15 - col. 4, lines 19). Matsumoto also discloses that the importance level 
can also be adjusted by the user operating the surveillance system {Col. 7, lines 22-34). 
Matsumoto et al. further discloses that by adjusting the compression of the video being 
captured, it is possible to record the monitoring image data having a high degree of 
importance as much as possible {Col. 7, lines 35-43). In col. 4, lines 4, lines 1-19, 
Matsumoto et al. further teaches that based on the degree of importance of the image 
data being recorded, when the memory has no residual capacity images that have lower 
level of importance would be identified and deleted or overwritten so that images with 
high degree of importance would be stored in the storage means (This teaches that the 
memory stores the image signal in place of a stored image when the value of the 
saliency signal is greater than a value of a second saliency signal associated with the 
stored image and the memory is full as claimed). 

Therefore, taking the combined teaching of Metcalfe in view of Takahashi and 
further in view of Matsumoto et al. as a whole, it would have been obvious to one of 
ordinary skill in the art a the time the invention was made to apply the concept of 
replacing images with lower level of importance with images captured with higher levels 
of importance when the memory has no residual capacity, images that have lower level 
of importance would be identified and deleted or overwritten as discussed in Matsumoto 
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et al. to modify tlie teaching of Metcalfe and Takahashi by having the memory storing 
the image signal in place of a stored image when the value of the saliency signal is 
greater than a value of a second saliency signal associated with the stored image and 
the memory is full. The motivation to do so would have been to record the monitoring 
image data having a high degree of importance as much as possible as suggested in 
Matsumoto et al. (Co/. 7, lines 35-43). 

29. Regarding claim 42, the combined teaching of Metcalfe in view of Takahashi 

and further in view of Matsumoto et al. as discussed and analyzed in claim 40 further 
teaches a separate user operable picture taking control for selectively activating the 
electronic camera to take pictures {As shown in Takaliaslii, figs. 4(b): 109; fig. 9(a): 109, 
10: 109a, and 11: 109b, Takahashi discloses the use of a plurality of buttons to select 
from different auxiliary information to be assigned to the video signal (note that the 
buttons 109 are physically located on the camera). Furthermore, as shown in figs 23(a) 
and 23(b), Takahashi further discloses that the auxiliary information can be selected 
using a monitor, wherein the user can select the person information and the degree of 
importance of the persons using buttons 103m, 103n, 101m and 101 n; page 8, H 0119 
(although these buttons are displayed on the touch screen display, the Examiner 
understands that the buttons are considered different user controls since they have 
different physical location on said display)). 
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Conclusion 

30. Applicant's amendment necessitated tine new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 
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